Dynamic Adaptive File Management in a Local Area Network

In light of advances in processor and networking technology, especially the emergenceof network attached disks, the traditional clientserver architecture of file systems has become suboptimal for many computation/data intensive applications. In this paper, we introduce a revised architecture for file management employing network attached storage: the dynamic file server environment (Dynamo). Dynamo introduces two main architectural innovations: (1) To provide high scalability, the file management functions are mainly performed cooperatively by the clients in the system. Furthermore, data is transferred directly to the client’s cache from network-attached disks, thus avoiding copies from a disk to the server buffer and then over the network to the client. (2) Dynamo uses a cooperative cache management which employs a decentralized lottery-based page replacement strategy. We show via performance benchmarks run on the Dynamo system and simulation results how this architecture increases the system’s adaptability, scalability and cost performance. 1 Overview of Dynamo The Dynamo architecture consists of four layers: the disk I/O layer, the cooperative cache manager layer, the file manager layer, and the coordinator layer. These four layers interact with each other as shown in Figure 1. Disk I/O Layer The lowest layer of Dynamo is the I/O layer that provides data I/O from and to storage devices. We assume that the storage devices are disks or disk arrays. The I/O layer resides on the disk controllers of the network attached storage devices and maps files and data pages to storage locations on a storage device in a similar fashion to the I/O layer in a conventional file system. Cooperative Cache Manager Layer In a LAN, the working set size of local applications will vary over time resulting in time when the working set of an individual node exceeds the node’s local physical memory; however, the aggregate size (i.e., of the union) of all working sets is often less than the aggregate nodes’ combined main memory. Therefore, it is beneficial if local working sets can “spill over” to other node whose main memory is not fully utilized. To achieve this, Dynamo treats main memory from all nodes as a pool of global memory with a cooperative cache manager layer on each node. Each node treats its own memory as the local cache and memory on other nodes as a remote cache, intermediate between its local cache and secondary storage. The cooperative cache managers are responsible for managing the remote caches as well as their own local caches. The cooperative cache manager (CCM) consists of two components: the local cache manager (LCM) and the distributed cache manager (DCM) which collaborates with other DCMs. The LCM performs cache replacement of its local cache while a DCM defines, in concert with other cache managers, a decentralized scheme for global cache management. Using a lottery-based scheme, the DCM determines how and whereto local pages should be evicted. CCM also maps between the logical address of a file, and its memory address. File Manager Layer In Dynamo, large parts of server functions are distributed to nodes in the LAN and are handled by them in a cooperative fashion. We distinguish two software components in Dynamo which provide the functionality of a traditional file system: the coordinator and the file managers. The coordinator is a small remaining part which runs on one or more well-know core nodes, while the file managers execute on any nodes and perform most of the traditional data management functions. We refer to the manager of a file as the owner of that file. Coordinator Layer The coordinator manages centralized information for all nodes, and coordinates which data partition is managed by which nodes, keeps track of which data is managed by which file manager, informs nodes about owners, and participates in ownership transfers between nodes as well as transaction management. Because of its central role, a coordinator (or a set of cooperating coordinators) run on a backbone of reliable, well-known nodes. In the following sections, we describe the contributionsand specifics of Dynamo such as the dynamic data management strategy and the decentralized cache management strategy for its cooperative cache in detail. Ownership of Global Ownership Table Maintenance Coordinator Core Machine Page-Block Mapping I/O Manager Disk Block Allocation Strategies Change Disk Controller Page Applications File-Pages Mapping Cache Table Lookup Local Ownership Table Maintenance File Manager Distributed Cache Manager Cache Table for Onwed Files Cooperative Cache Manager Local Cache Manager Local Cache Maintenance Applications Distributed Cache Manager Cache Table for Onwed Files Cooperative Cache Manager Local Cache Manager Local Cache Maintenance File-Pages Mapping Cache Table Lookup Local Ownership Table Maintenance File Manager Machine Machine Figure 1: Dynamo Architecture